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Abstract 

In this paper we introduce a novel platform for dense 3D modelling. 
This platform is an active image acquisition setup assisted with a set of 
light sources and a distance sensor. The hardware setup is designed for be¬ 
ing mounted on a mobile robot which is remotely driven to create accurate 
dense 3D models from out-of-reach objects. For this reason, the object is 
actively illuminated by the imaging setup and Photometric Stereo is used 
to recover the dense 3D model. The proposed image acquisition setup, 
called LightPanel, is described from design to calibration and discusses 
the practical challenges of using Photometric Stereo under uncontrolled 
lighting conditions. 


1 Introduction 

This paper presents a practical system for using Photometric Stereo (PS) in 
robotics applications. The hardware consists of a configurable image acquisi¬ 
tion system, designed to be mounted on a mobile robot, i.e. KUKA youBot PQ. 
The motivation of using such 3D reconstruction system is to accurately create 
3D models of objects located in some places which are either dangerous or dif¬ 
ficult for humans to reach in search and rescue scenarios. Resulting dense 3D 
models can be used for critical tasks where the objects have to be manipulated 
remotely, i.e. inspection, part insertion, drilling, and so on. The LightPanel 
illuminates the object from different directions and takes images under vary¬ 
ing lighting conditions from a fixed view point. Such images are suitable for 
photometric dense reconstruction using PS. LightPanel can be easily tuned by 
changing the distance and angle of each light source with respect to the cam¬ 
era. Reconfigurability of the LightPanel allows illuminating objects in different 
distances for dense 3D reconstruction. Fig. [I] shows the LightPanel and its 
components mounted on the KUKA youBot. 

PS is a technique to estimate the surface points’ orientations from multiple 
images under varying illumination, introduced by Woodham in j2] . The Wood- 
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Figure 1: The LightPanel mounted on a KUKA youBot. 


ham’s algorithm [2] has been enhanced in different directions. Uncalibrated PS, 
for unknown lighting was introduced by Hayakawa [3] and improved in B9 IS Si- 
Further work to extend the algorithm with more complex lighting models for 
non-diffusive materials is highlighted in mmm- Another way of improving the 
standard calibrated photometric stereo algorithm was presented in [13 EOT 
by adapting the formulas to a perspective camera model instead of the ortho¬ 
graphic projection. In E3EI1E3] the assumption of a distant light source is 
relaxed. The field of application is wide and reaches from industrial inspection 
pff) and quality control m to archaeology with studies of ancient artefacts [ 16 ]. 
In D3i a dome was introduced, where the object is placed in the centre and 
the light sources are fixed on the walls of the dome. A frame for calibrating 
the illumination directions is placed around the object in [16] with a hand-held 
light source. In m a big setup has to be installed in front of the object and 
in [HIES] the object has to be placed in a capturing tool. Most similar image 
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acquisition designs to our LightPanel are introduced by Zhou m and Malzben- 
der [20j where the first real-time PS algorithm was implemented with a setup 
using wings equipped with LEDs. 

Although PS reconstruction and its application is widely studied in the lit¬ 
erature, the lighting condition is either controlled or a priori knowledge is con¬ 
sidered in this regard. The LightPanel hardware is low-cost and energy efficient 
which makes it suitable for using on a mobile robot. This work investigates 
the practical challenges in employing PS reconstruction in environments with 
uncontrolled lighting condition for robotics applications. In addition to intro¬ 
ducing a mobile image acquisition system, this paper looks into two fundamental 
questions regarding the lighting condition: 

• How does the active illumination affect the reconstruction? How should 
the object be illuminated? 

• Is the ambient light helpful for PS reconstruction or is not desirable at 
all? 

In the following section, a brief introduction to photometric image formation 
and PS dense reconstruction is given. Afterwards, the proposed image acqui¬ 
sition hardware (i.e. LightPanel) is introduced and supported by discussion 
on calibration the system. Then, practical challenges to use LightPanel are 
discussed and followed by the reconstruction results. 

2 Photometric Stereo 

Standard PS methods assume that the camera and object are fixed and a single 
distant light source is moved during the image acquisition. For this reason, the 
correspondence between the image points is given and can also be used for fea¬ 
tureless objects [21] ■ Since no features have to be matched, it is computationally 
cheaper than other reconstruction methods. Assuming a perfect defuse surface, 
the observed intensity I(x,y) of a surface point s is defined as: 

I(x,y) =pcos(i) = ^_ l( 1+ Pk +g^) (1) 

V i +p 2 + g 2 y1 + 11 + 11 

where i is the light incident angle and p is the reflectance factor, also called 
albedo which is defined as diffuse reflectivity, p , q , l x and l y are the components 
of vectors representing the surface normal and the light source direction, as de¬ 
picted in Fig. [2] In ([T]) the observed intensity does not depend on the viewer’s 
direction. Such relation is only valid for perfect diffuse surfaces, called Lam¬ 
bertian surfaces, which assumes that the incident light is equally reflected in all 
directions [22| . 

Given / images and the light source directions, the surface normal vector and 
albedo values can be estimated for all the observed surface points [2] . Therefore, 
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Figure 2: Reflectance geometry: Incident angle i is between the light source ray 
and the surface normal and the phase angle g is between the light source ray 
and the viewer direction. 

0 can be rewritten in matrix form as: 

I = Lnp, (2) 

where matrix I/xb is the stack of / vectorized images, each containing b pixels. 
Matrices L / X 3 and n 3 X b contain the lighting direction and surface normal vec¬ 
tors, respectively. The diagonal matrix pbxb contains the albedo values for all 
the pixels. In case of calibrated PS—with known light source directions—three 
images are enough to estimate the surface normals. Because of shadows and 
not perfect Lambertian properties as described by Sun in [S], three images are 
mostly not enough when working with real data. Therefore, / > 3. Given the 
light source directions L and the intensity matrix I, albedos and surface normals 
can be calculated as: 


drag (|L 1 I ), 

(3) 

1 , 

n = —ID 1 !. 

(4) 


P 


3 Designing the LightPanel 

Our image acquisition system, the LightPanel , consists of a camera, a distance 
sensor and several light sources—called LightBlob —on the circumference of a 
circle around the camera. Every LightBlob has 2-DOF, which are used to change 
their distance from the camera and change the emitting angle. Using such 
configurable LightBlobs allow the LightPanel to have different emitting focal 
length and consequently be able to capture images from objects at different 
distances from the camera. Every LightBlob consists of several LEDs assembled 
in a way to provide proper lighting to the scene. 
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Figure 3: Arrangement of LEDs in the LightBlob. 


3.1 LightBlob 

The LightBlob is an arrangement of 19 LEDs with viewing angles of 15° sitting 
together to act as single light source that has a wider viewing angle and uni¬ 
formly illuminates the scene. Such arrangement satisfies the standard calibrated 
PS algorithm assumption of parallel light rays from a single distant light source. 
The advantage of a small opening angle is that the light direction is defined by 
the orientation of LEDs, but indeed the covered area is smaller. Therefore, in 
every LightBlob multiple bright LEDs are arranged in a way that they jointly 
build a cone and cover a bigger area. The luminous intensity of a single LED is 
35 Cd at a current of 20 toA [231 . The relative luminous intensity depending on 
the forward current as well as the radiation properties are considered to find the 
angles between LEDs to achieve a smooth intensity on the object. Considering 
the LEDs specifications the angles between the LEDs is defined to be 11.25°, 
as it is shown in the Figure [3] The electronic circuit that derives the LightBlob 
allow for vary its viewing angle between 30° and 52.5°, by turning the outer 
ring of LEDs on and off. This allows to have parallel light rays even at close 
distances. 

3.2 Light Panel 

The main body of the LightPanle is a disc with the camera in the middle. 
Around the disc, several wings, each having a LightBlobs , are attached. The 
design allows adding up to twelve wings. Every wing consists of spacers and a 
joint to align the LightBlobs with a certain focal point. Using the joint, the angle 
of light direction can be varied from 10° to 80° in steps of 10°. Combination of 
the length of different spacers and the joint angles define the focal point of the 
light panel. The distance between camera and the focal point is the operating 
distance of the light panel and is called d 0 t,j ec t- The schematic of such a distance 
is illustrated in Fig. [4] 

The distance d 0 bj e ct is defined by the phase angle g and the distances g?i and 
o? 2 , where d\ = d r + d spaC er + dj 0 i n t is the sum of the disc radius d r , which holds 
the camera and the wings, a variable spacer d spa cer in the middle and the joint 
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Figure 4: Illustration of a wing in the LightPanel. 

radius dj 0 i n t . Similarly, d 2 = djomt + d spacer + dbiob consists of the joint radius 
djoint-, a part of the light blob dbiob and a variable spacer d spacer . The distance 
dobject is given by 

, _ d\ d 2 , , 

object tan (g) sin (g )' 

Therefore, the light source direction l s = [— sin(g), 0, cos(<?)] T depends only on 
the phase angle g. 

3.3 Calibration 

The aim of calibration process is to estimate the light source directions in the 
camera frame. For this reason the camera frame C has to be estimated in the 
first step. The light panel P is equipped with markers to track the light panel 
position in the motion capture system Hop. Calibration of the camera provides 
the checkerboard position in the camera frame H cb- Using Matlab calibration 
toolbox [23], the transformation Hpc is obtained. Then, the light direction 
vector for every Light-Blob U (i = 1.../), has to be derived. Such vectors 
can be obtained by considering the LightPanel geometry, so the transformation 
of light vectors from the light frames to the world reference frame are called 
UoLi ■ Therefore, the transformation of the light frames to the camera frame, as 
illustrated in Fig. [5] can be defined as: 

H CLi = (HopHpc) _1 HoL i . (6) 


4 Experiments and Discussions 

In this section challenges in designing and using an image acquisition setup are 
discussed and followed by a few results obtained by using the LightPanel in 


6 







Figure 5: Transformations of the LightPanel. The unknown transformation 
H cLi is illustrated in magenta. 


practical applications are presented. 

4.1 How should the light be illuminated? 

In order to investigate the effect of light emitting direction on the reconstruction 
an experiment is designed in a simulating environment. More specifically, the 
goal of this experiment is to find what is the best value for phase angle g 1 which 
is introduced in both Fig. [2] and Fig. [4] A synthetic unit sphere with perfect 
diffuse surface is created and an orthographic camera is placed in front of it. 
The light source is chosen to have parallel rays pointing to the centre of the 
sphere. The intensity of the light source was chosen in a way that no pixel 
intensity is saturated. The light source is rotated 360° around the camera and 
every 60° an image was rendered. The number of images was increased from 
three images of the standard calibrated photometric stereo algorithm to six 
images, because with six images of a convex object illuminated by six different 
light positions it can be ensured that every visible surface point is illuminated 
by three light sources [9]. In this setup the only parameter is the phase angle 
g. Therefore, the phase angle was varied from 1° to 89° in steps of 1°. Using 
([3| and Q on synthetic data consisting of six images of a diffuse sphere with 
known illumination and a constant phase angle g , the estimated normals are 
compared with the normals of the sphere. The mean and median relative errors 
are shown in Fig. [6] 
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Figure 6: Mean and Median reconstruction error for the surface normals. 


4.2 Is the ambient light disturbing? 

The LightPcinel should be applicable in the presence of ambient light. To effec¬ 
tively estimate the surface normals, the ambient light should be removed from 
the images. Therefore, as suggested by m, for using the LightPanel an image 
is taken without turning on the LightBlobs. Then, this image is subtracted from 
images taken under active lighting. To investigate the effect of ambient light, a 
light source which imitates the ambient light is added to the synthetic experi¬ 
ment. The amount of ambient light is measured with the pixel intensities on a 
scale from 0 to 255 of the sphere in the ambient light image. The comparison 
of the estimations proceeds as in the experiments before. Fig. [7] illustrates the 
mean and median relative errors against the mean ambient intensity. The best 
result of the four datasets is achieved with about 20% ambient light, which shows 
that the light panel can be used with ambient light. It means that having some 
ambient light helps the active light source to illuminate the object smoothly by 
decreasing the number of shadowed pixels. However, too much ambient light 
can disturb the whole estimation if the pixel intensities are saturated with the 
ambient light. 

4.3 Results 

Several experiments have been done using the LightPanel to reconstruct objects 
and scenes® The camera captures 752 x 480 pixel grayscale images at 12 Hz. 
Fig-® shows one of the most challenging scenarios used for reconstruction. For 
this experiment, the phase angle g = 30° and distance d 0 bj e ct = 0.35m and 
images are captured with 6 different lighting conditions. In this experiment, the 

1 More results can be found in this video 
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Figure 7: Mean and median relative error with increasing ambient light. 


(a) Scene with ambient light 


(b) Recovered albedo 



(c) Recovered surface (d) Recovered normal map 

Figure 8: Reconstruction of scene with multiple objects. 


objects are not segmented from the background and the resulted normal map 
contains many outliers for the surface points belonging to the background. Such 
artefacts in the normal map make the normal integration step—to get surface 
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from normal map—more challenging. Consequently, the reconstruction of the 
normal map in Matlab took 14.58 Sec for 153756 Pixels. A further problem is 
the occlusion of the objects in the background, because the occluded points are 
not sufficiently illuminated. A possible solution to improve this reconstruction 
could be to equip the panel with more light sources and try different algorithms 
for the segmentation of the objects before applying photometric stereo. 

5 Conclusions 

This paper introduces a mobile setup which uses active light and employs PS for 
dense 3D modelling. Operating this system in uncontrolled lighting conditions 
introduces challenges for using PS in practice. Such challenges are addressed 
in this paper and supported by experiments on synthetic and real data. In 
a nutshell, we confirm that the ambient light can help PS reconstruction by 
reducing the number of shadowed pixel and providing a smooth normal map. 
Moreover, the incident angle of light rays plays an important role in the quality 
of reconstruction. Our experiments shows that the best results can be achieved 
by setting this angle between 20° and 30°. 
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